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Abstract 

Robertson's 2-poisson information retrieve model does not use location and category information. We 
constructed a framework using location and category information in a 2-poisson model. We submitted 
two systems based on this framework to the IREX contest, Japanese language information retrieval 
contest held in Japan in 1999. For precision in the A-judgement measure they scored 0.4926 and 
0.4827, the highest values among the 15 teams and 22 systems that participated in the IREX contest. 
We describe our systems and the comparative experiments done when various parameters were 
changed. These experiments confirmed the effectiveness of using location and category information. 
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1 Introduction 

Information retrieval (IR) has become an increasingly im- 
portant area of research due to the rapid growth of the 
Internet. In 1999 the Information Retrieval and Extrac- 
tion Exercise contest (IREX) was held in Japan. We 
submitted two systems to this contest. Their precision 
in the A-judgement measure^] was 0.4926 and 0.4827, the 
highest values among the 15 teams and 22 systems in the 
IREX contest. This paper describes our systems and the 
comparative experiments done when various parameters 
were changed. 

Our information retrieval method uses Robertson's 2- 
poisson model H, which is one kind of probabilistic ap- 
proach. But, Robertson's method does not use location 
or category information, which should be used to facili- 
tate information retrieval. Against this background, we 
constructed a framework by using location information, 
category information, and detailed information in a 2- 
poisson mode^]. We verified the effectiveness of using 

1 A-judgmcnt means that a document whose topic is relevant 
to a query is judged a relevant document. 

2 The reason that this paper is entitled "Probabilistic Infor- 
mation Retrieval Using Location Information and Category Infor- 
mation" is that our methods use location and category informa- 
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these three types of information by doing comparative 
experiments. When the 2-poisson model is used a term 
extraction method needs to be selected. In this paper, 
we describe four term extraction methods, and compared 
them in experiments. 

2 Information retrieval 
2.1 Task 

The information retrieval tasks in this paper are identical 
to those for the IREX contest. The database used for 
information retrieval (the same used in IREX) is from 
two-years (1994-1995) of a Japanese newspaper ||. We 
retrieved from this database documents which satisfied 
the information condition for a Japanese language query. 

The following is an example of a query. (The data is 
from the IREX preliminary experiment.) 

Example of a Japanese query 

<T0PIC> 

<T0PIC-ID>100K/T0PIC-ID> 
<DESCRIPTIONX/DESCRIPTION> 
<NARRATIVEX/NARRATIVE> 
</T0PIC> 

English translation 
<T0PIC> 

<TOPIC-ID>100K/TOPIC-ID> 
<DESCRIPTION>enterprise amalgamation 
</DESCRIPTION> 
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tion in addition to Robertson's probabilistic information retrieval 
method. 



<NARRATIVE> 

The condition for a relevent document is 
that in the document an announcement of 
enterprise amalgamation materialization is 
described and the name of the enterprise 
which participated the amalgamation can be 
recognized. Also, one of the field of 
amalgamation enterprise and its purpose 
should be able to recognized. Enterprise 
amalgamation contains enterprise annexation, 
enterprise integration, and enterprise 
purchasing. </NARRATIVE> 
</T0PIC> 

The number indicated by <T0PIC-ID> means the ID 
number of the query. <DESCRIPTION> contains a phrase 
that indicates the information needed. <NARRATIVE> 
contains the sentences that restrict the information re- 
quested. During the task, the system receives a query 
such as the above one and outputs 300 documents in 
order of confidence. 

Example of a Japanese document 

<DQCND>95021709K/DDCNQ> 
<SECTI0NX/SECTI0N> 
<HEADLINEX/HEADLINE> 
<TEXT> 



score using the following equation^], then outputs the doc- 
uments with high scores as retrieval results. (The follow- 
ing Score(d, q) is the score of a document d against a 
query q.) 



Score(d, q) — ^ 



term t \ */(<*, t) + *t 
in q 



tf(d,t) io N 

length(d) ° 9 df(t) 



tf q (q,t) 
tf q (q,t) + kq 



(1) 



where terms occur in a query. tf(d, t) is the frequency of 
a term t in a document d, tf q (q, t) is the frequency of t in 
a query q, df(t) is the number of the documents in which 
t occurs, TV" is the total number of documents, length(d) 
is the length of a document d, and A is the average length 
of the documents. k t and k q are constants which are set 
by experiments. 



In this equation, we call 



tf(d,t) + k t 



length{d) 
A 



the TF 



term, (abbr. TF(d,t)), log-^ the IDF term, (abbr. 
IDFit)), and tf tf { q q ( ^ kq the TF,j term, (abbr. TF q (q,t)). 

Our method adds several extended terms to this equa- 
tion, and is expressed by the following equation. 



</TEXT> 
</D0C> 

English translation 

<DQCND>95021709K/DDCNQ> 
<SECTION>Economic page</SECTIDN> 
<HEADLINE>Tounen company completely makes 
Kigunasu company a subsidiary company 
</HEADLINE> 
<TEXT> 

Tounen company anounced that it makes 
Kigunasu comapry, one of its group companies, 
(The capital is one billion yen. The head 
office is in Kawasaki city. The president 
is Mr. Toshihide Mori.) a completely 
subsidiary company. Kigunasu company is 
invested 70°/, by Tounen company and 30°/, by 
Nichimou company, and Tounen company 
purchases 600,000 stocks which Nichimou 
company possesses at 12,500,000,000 yens. 
</TEXT> 
</D0C> 

In this document, the newspaper information cate- 
gory (the economic or political pages) is indicated by 
<SECTI0N>, the title of the document is indicated by 
<HEADLINE>, and the body of the document is indicated 
by <TEXT>. The tool "trec.eval" of TREC is used to 



evaluate the retrieval results 
Precision" were used. It indicat 
trieving R documents, where R is the number of relevant 
documents. 



In the contest, "R- 
es the precision when re- 



2.2 Outline of our method 

Our information retrieval method uses Robertson's 2- 
poisson model which is one kind of probabilistic ap- 
proach. Robertson's method calculates each document's 



S,,, , ,,; ,/.,, i = Kcategoryid) < ^ (TF(d,t) IDF(t) 

term t 
in q 

TF q (q,t) K dctaU {d,t) K locatiorl (d,t)) 
length(d) 



length(d) + 



(2) 



The TF, IDF and TF q terms in this equation are 
identical to those in Eq. (yj). The term ie '^ t ^ A has 
a higher value when a document is longer. This term is 
made because if the other information is exactly equal, 
the longer document is more likely to include the content 
requested by the query. K categ ory, K dctal i and K io 

cation 

are extended numerical terms made to improve precision. 
K ca tegory uses the category information of the document 
found in newspapers, such as the economic and political 
pages. Ki ocation uses the location of the term in the doc- 
ument. If a term is in the title or at the beginning of 
the body of the document, it is given a higher weighting. 
Kdetaii uses the information such as whether the term is a 
proper noun and or a stop word such as "document" and 
"thing". In the next section, we explain these extended 
numerical terms in detail. 

2.3 Extended numerical terms 

We use the three extended numerical terms of Ki ocation , 



K. 



category , 



and Kdetaii as in Eq. (E2|) . This section explains 



them in detail. 



1. Location information (Ki ocat ion) 



In general, the title or the first sentence of the body 
of a document in a newspaper very often indicates 
its subject. Therefore, the precision of information 



This equatjipn is BM11, which corresponds to BM25 in the 
case of b — Although we made experiments testing some 

cases of b in BM25, the case of b — 1 was roughly better than any 
other cases in this work. So we used BM11. 



retrieval can be improved by weighting the terms in 
these two locations. The term Ki oca uon performs 
this task, and changes the weight of a term based 
its location at the beginning of the document. If a 
term is in the title or at the beginning of the body, 
it is given a high weighting. Otherwise, it is given 
low weighting. This term is expressed as follows: 



K locaU on{d,t) 



(when a term t occurs in the title of 
a document d), 



1 + k locatio 
(otherwise) 



(length(d) - 2 * P{d,t)) 
length(d) 



(3) 



P{d, i) is the location where a term t occurs in the 
document d. When a term occurs more than once in 
a document, its first occurrence is used. ki oca tion,i 
and ki oca tion,2 are constants which are set by ex- 
periments. 

2. Category information (Kcategory) 

Kcategory uses category information such as the 
economic and political pages. This functions as a 
technique called relevance feedback [ju^. First, we 
specify the categories which occur in the top 100 
documents of the first retrieval when K category = 1- 
Then, we increase the scores of documents hav- 
ing the same categories. For example, if economic 
pages often occur in the top 100 documents of 
the first retrieval, we increase the score of a docu- 
ment whose page is a economic page and decrease 
the score of the document whose page is different. 
K C ate S ory is expressed as follows; 



category v^J — 1 ~\~ ^category 



(RatioA(d) - RatioB(d)) 



(RatioA(d) + RatioB(d)) 
(4) 

where RatioA is the ratio of a category in the top 
100 documents of the first retrieval. RatioB is the 
ratio of a category in all the documents. The value 
of Kcategory (d) is large, when RatioA is large (page 
of a document d occurs frequently in the top 100 
documents of the first retrieval.) and RatioB is 
small (page of a document d does not occur often 
in all the documents.), kcategory is a constant which 
is set by experiments. 

3. Other information (Kdetaii) 



KdetaU is a more detailed numerical term that uses 
different information, such as whether the term is 
a proper noun and whether the term is a stop word 
such as "document" and "thing". If a term is 
a proper noun, it is weighted high. If a term is a 
stop word, such as "document" and "thing," it 
is weighted low. Kdetaii is expressed as follows for 
simplicity, the variables for a document and a term, 
d and t, are omitted: 



K de 



K de 
K hi 



zr K p r p er K ' n 



(5) 



Each term in this equation is explained below. 



Kdescr 

When a term is obtained from the title of a 
query, i.e. DESCRIPTION, K desoT = k de3cr (> 
1). Otherwise, Kdescr = 1. This is because 
a term obtained from the title of a query is 
important. 

Kproper 

When a term is a proper noun, K vroV er = 
k P roper(> 1)- Otherwise K pr0 per = 1. This 
is because a term that is a proper noun is im- 
portant. 

Knado 

When a term is followed by the Japanese word 
nado (such as) in a query sentence, K nado = 
k„ado(> 1). Otherwise K„ ado = 1. A term 
which is followed by the Japanese word nado 
is specific in meaning and is just as important 
as a proper noun. 

Knum 

When a term is numeric, K num — k num (< 1). 
Otherwise, Knum = 1. A term which consists 
of only numerals does not contain much rele- 
vant information making it unimportant to a 
query. 

Kfiira 

When a term consists of hiragana characters 
only, Khira = k h ira{< 1)- Otherwise, K ni ra 
= 1. A term which consists of only hiragana 
characters does not contain much relevant in- 
formation making it unimportant to a query. 

Kneg 

When a term is obtained from a region tagged 
with a NEG tag in a query, K„e 3 = k neg . Oth- 
erwise K n eg = 1- 

In a query of the IREX contest, an expression, 
"... wa nozoku" (... is excepted), as in the 
following query, was tagged with a NEG tag. 

Example Japanese query 
<T0PIC> 

<T0PIC-ID>1003</T0PIC-ID> 
<DESCRIPTI0NX/DESCRIPTI0N> 
<NARRATIVEXNEG></NEGX/NARRATIVE> 
</T0PIC> 

English translation 
<TQPIC> 

<T0PIC-ID>1003</T0PIC-ID> 
<DESCRIPTI0N>Dispatch of the United 
Nations f orces</DESCRIPTI0N> 
<NARRATIVE>The condition for a 
relevent document is that in the 
document a dispatch of the United 
Nations forces in the activity of UN 
such as peace maintainence activity 
is described. The purpose of the 
dispatch or the target region should 
be described. <NEG>A document 
describing the discussion of whether 
the Self-Defense Forces of Japan is 
dispatched to UN or not is elimated. 
< /NEGX /NARRATT VE> 
</T0PIC> 

If a term from a region tagged with a NEG tag 
is used, non-relevant documents are often re- 
trieved and therefore such a term is weighted 



low. In this paper, k neg is set to 0. This indi- 
cates that a term from a region tagged with a 
NEG tag is not used in retrieval. 

• K 

stopword 

When a term is a stopword such as jouken 
(condition), kiji (document) and baai (case), 
K 

stopword — ksto-pwordiy*^ 1). Otherwise A" stopword 

= 1. A term that is a stopword is unimportant. 
Each constant, such as kdescr, is set experimentally. 



2.4 How to extract terms 

Before being able to use Eq. (j^ in information retrieval, 
we must extract the terms from a query. This section 
describes how to do this. With regard to term extraction, 
we considered the several methods listed below. 

1. Method using only the shortest terms 

This is the simplest method. The method divides 
the query sentence into short terms by using the 
morphological analyzer "juman" Hland eliminates 
non-nominal words and stop wordsn. The remain- 
ing words are used in the retrieval process. 

2. Method using all term patterns 

In the first method the terms are too short. For 
example, "enterprise" and "amalgamation" are 
used instead of "enterprise amalgamation."^] We 
thought that we should use "enterprise amalgama- 
tion" in addition to the two short terms. Therefore, 
we decided to use both short and long terms. We 
call this "all-term patterns method." For example, 
when "enterprise amalgamation materialization" 
was inputted, we use "enterprise", "amalgama- 
tion", "materialization", "enterprise amalgama- 
tion", "amalgamation materialization", and "en- 
terprise amalgamation materialization" as terms for 
information retrieval. We thought that this method 
would be effective because it uses all term patterns. 
But, we also thought that it is inequitable that only 
the three terms of "enterprise," "amalgamation," 
"materialization," are derived from "... enterprise 
... amalgamation ... materialization while on 
the other hand six terms are derived from "en- 
terprise amalgamation materialization." We exam- 
ined several normalization methods in preliminary 
experiments, and decided to divide the weight of 

each term by \J n< ~ n ^~ 1 \ where n is the number of 

successive words. For example, in the case of "en- 
terprise amalgamation materialization", n = 3. 

3. Method using a lattice 

Although the method using all-term patterns is ef- 
fective for use with all patterns of terms, it needs to 

be normalized by using the adhoc equation \J n ^ n ^~ 1 ^ - 

Thus, we considered the method where all the term 
patterns are stored into a lattice structure. We use 
the patterns in the path where the score in Eq. 

4 Since, Japanese is an agglutinative languages like Chinese, 
there are no spaces between words and a morphological analyzer 
is necessary to divide a sentence into words. 

5 Although this paper deals only with Japanese, not English, for 
this explanation we use English examples for the English readers. 
This method handles compound nouns and can be used not only 
for Japanese but also for English. 



Figure 1: An example of a lattice structure 



(|2|) is the highest. (This method is almost same as 
Ozawa's M. The differences are the fundamental 
equation for information retrieval, and whether to 
use or not use a morphological analyzer.) 

For example, in the case of "enterprise amalgama- 
tion materialization" a lattice, as shown in Fig. 
[I], is obtained. As in this figure, four paths exist 
where each of their scores are calculated by Eq. (0) 
and the terms in the highest path are used. This 
method does not require the adhoc normalization 
as in the method using all the term patterns. 

4. Method using down- weighting Q 

This is the method that Fujita proposed at the 
IREX contest, and we examined after the contest. 
It is similar to the all-term patterns method. It 
uses all the term patterns but the normalization 
is different from the all-term patterns method. It 
does not change the weight of the shortest terms; 
and decreases the weight of the longer terms. We 
decided to multiply the weight kd own x ~ x to a term, 
when it consisted of x shortest terms, where kdown 
was set by experiments. This method basically uses 
the shortest terms while also using the longer terms 
by down-weighting them. 

3 IREX contest results 

For our two submissions to the IREX contestQ we se- 
lected the "all-term patterns" and "lattice structure" 
methods to extract termsQ and set the constants of the 
extended terms in order to maximize the precision in the 
preliminary-run data as follows. 

1. System A 

It used the lattice method for the term extraction. 
The parameters were set as follows; ki oca tion,i = 

1.35, kl OC ation.2 — 0.125, kcategory — 0, kdescr — 1.5, 
kproper — 2, knado — 1, k n um — 0.5, khira — 0.5, 
kneg = 0, k s topword,l = 0, k s topword,2 = 0.5, kt = 1, 

and k q = 0.1. Terms obtained from DESCRIP- 
TION are handled as terms different from terms 
obtained from NARRATIVE. 

2. System B 

6 IREX allowed two systems to be submitted. 

7 The reason we did not use "the shortest terms method" is 
because it is too simple and did not seem effective. The "down- 
weighting method" is a method proposed at IREX. So we could 
not use it in IREX. 



Table 1: R-Precision of all the systems 



System ID 


A- Judgment 


B- Judgment 


1103a 


0.4505 


0.4888 


1103b 


0.4657 


0.5201 


1106 


0.2360 


0.2120 


1110 


0.3329 


0.4276 


1112 


0.2790 


0.3343 


1120 


0.2713 


0.3339 


1122a 


0.3808 


0.4689 


1122b 


0.4034 


0.4747 


1126 


0.0966 


0.0891 


1128a 


0.3384 


0.3897 


1128b 


0.3924 


0.4175 


1132 


0.0602 


0.0791 


1133a 


0.2383 


0.2277 


1133b 


0.2457 


0.2248 


1135a 


0.4926 


0.5119 


1135b 


0.4827 


0.4878 


1142 


0.4455 


0.4929 


1144a 


0.4658 


0.5510 


1144b 


0.4592 


0.5442 


1145a 


0.3352 


0.3424 


1145b 


0.2553 


0.2935 


1146 


0.2220 


0.2742 



It used all-term patterns method for term extrac- 
tion. The parameters were set as follows; ki OC ation,i 

1.3, ^location^l — 0.15, k ca tegory — 0.1, ^descr 



A.'n 



— 0, k s t 



opword,! 



— 0, k s to 



pword,2 



= 0.5, 



0.5, 

k t = 1, and k q — 0. Terms obtained from DE- 
SCRIPTION were handled as the terms different 
from those obtained from NARRATIVE. 

In the contest, the results for the 22 systems were sub- 
mitted bv the 15 teams. Their R-Precisions are shown 
in Table pj. The first column of the table indicates the 
names of the systems. Our two systems, System A and 
System B correspond to 1135a and 1135b. A- Judgement 
and B- Judgement are the evaluation criteria determined 
by the IREX committee. A- Judgment means that a doc- 
ument whose topic is relevant to a query is judged as a 
relevant document. B- Judgment means that a document 
whose topic is partly relevant to a query is also judged as 
a relevant document. Although our systems were not the 
highest in B- Judgment, they were the highest among all 
the systems in A- Judgment. This result indicates that 
our method is relatively superior. 

4 Experiments 

In this section we describe several experiments done to 
test the effectiveness of the several methods used in our 
system. In the experimental results of this section, we 
also show Average Precision (the average of the precision 
when each relevant document is retrieved) in addition to 
R-Precision. For the comparison experiments, t-test is 
used. A method tagged with in Tables ti to rn) is 
the base for comparison. A method tagged with "*" is 
superior to the base method at the significance level of 
5%, and a method tagged with "**" is superior at the 
significance level of 1%. T-test is used only in formal- 
run experiments. (The preliminary-run data contained 
six queries, and the formal-run data contained thirteen 
queries.) 



4.1 Comparison of term extraction methods 

We showe d th e following four term extraction methods 
in Section 



2.4 



1. Method using the shortest terms 

2. Method using all the term patterns 

3. Method using a lattice 

4. Method using down-weighting 

All the comparison results are shown in Table E. In 
Table ^(a) all extended terms were used. In Table~[|b) 
no extended terms were used. In the down-weighting 
method we tested the two cases of kdown = 0.1 and 

kdown = 0.01. 

The precision of the all-term patterns method was 
lowest in the formal run. It needed to be normalized us- 
ing the adhoc equation. Since it had the lowest precision, 
it was thought to be inferior to the other methods. Also, 
it was shown by t-test to be significantly inferior to the 
shortest terms method. 

Although the down-weighting method obtained the 
highest precision when no extended terms were used, it 
was not as effective when all the extended terms were 
used. Since it was significantly different from any of the 
other methods, cannot say that it is very reliable. But, 
in the case where a small amount of retrieval information 
was used (i.e. no extended terms) it was very effective. 

Since only the shortest terms method is significantly 
different from the all-term patterns method, we think it is 
a sound method which can provide reliable results. Since 
the lattice and the down-weighting methods are not sig- 
nificantly different from the all-term patterns method, we 
think that they must have some problems. One problem 
that occurred when using the lattice method was that 
the terms used in retrieval easily changed depending on 
the context, while the down-weighting method's problem 
was that it uses the extra terms even if it down-weights 
them. However, it is thought by us that using longer 
terms in addition, is better than using only the short- 
est terms. We have to continue the investigation of term 
extractions. 

4.2 Effectiveness of extended terms 

Extended terms used in this paper are classified into the 
following three categories: 



1- Ki oca tion (location information) 

2. K C ategory (category information) 

3. Kdetaii (detail information) 
(Here, Kdetaii contains Ki engt h = 



length 



which 



length-\-A. 

is the numerical term for a document length in Eq 

§•) 

In order to verify the effectiveness of the above three 
extended terms, we carried out eight experiments in 
which these three terms were alternately used or not 
used. These experiments were performed using "the lat- 
tice method" and "the shortest terms method". The 
results are shown in Table ^. 

The last line of the table is the case where no extended 
terms were used and the first line of the table is the case 
where they were all used. When we compared the two 
lines, we found there was an improvement of 0.027 to 



Table 2: Comparison of methods to extract keywords 

(a) When all extended terms are used 





Formal run 


Preliminary run 


Method to extract terms 


R-Prccision 


Average 


3rccision 


H-Prccision 


Average precision 




A-Judgc 


H- Judge 


A-Judgc 


B- Judge 


A-Judgc 


B- Judge 


A-Judgc 


B- Judge 


Using the shortest terms 


0.5012 


0.5205" 


0.4935" 


0.4764* 


0.4412 


0.5442 


0.4546 


0.5151 


Using all term patterns* 


0.4827 


0.4878 


0.4553 


0.4453 


0.4373 


0.5573 


0.4576 


0.5317 


Using the lattice structure 


0.4926 


0.5119 


0.4808 


0.4698 


0.4599 


0.5499 


0.4638 


0.5170 


Using down-weight (kdown — 0.01) 


0.5006 


0.5217 


0.4935 


0.4778 


0.4412 


0.5445 


0.4546 


0.5157 


Using down-weight {k^ own — 0.1) 


0.4997 


0.5233 


0.4939 


0.4809 


0.4478 


0.5504 


0.4563 


0.5185 



(b) When no extended terms are used 





Formal run 


Preliminary run 


Method to extract terms 


R-Prccision 


Average 


precision 


R-Prccision 


Average 


precision 




A- J udgc 


B- J udgc 


A-Judgc 


B- J udgc 


A-Judgc 


B- J udgc 


A-Judgc 


B- J udgc 


Using the shortest terms 


0.4744 


0.4897 


0.4488* 


0.4487* 


0.3900 


0.5082 


0.3850 


0.4468 


Using all term patterns* 


0.4445 


0.4665 


0.4172 


0.4180 


0.3965 


0.4981 


0.3960 


0.4444 


Using the lattice structure 


0.4711 


0.4884 


0.4436 


0.4448 


0.4009 


0.5069 


0.3884 


0.4469 


Using down-weight (kdown — 0.01) 


0.4760 


0.4896 


0.4492 


0.4494 


0.3940 


0.5082 


0.3850 


0.4470 


Using down-weight (kdown — 0.1) 


0.4816 


0.4986 


0.4545 


0.4568 


0.4003 


0.5076 


0.3860 


0.4498 



The method tagged with is a base method for comparison. A result tagged with is superior to the base 



method's at the significance level of 5%, and a result tagged with is superior at the significance level of 1%. 



Table 3: Comparison of extended numerical terms 
(a) Comparison when using the lattice method 



Numerical terms 


Formal run 


Preliminary run 


R-Prccision 


Average precision 


R-Prccision 


Average precision 


-^■location 


^category 


^ detail 


A-Judge 


B- J udgc 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


yes 


yes 


yes 


0.5031 


0.5161 


0.4888* 


0.4745 


0.4495 


0.5471 


0.4625 


0.5202 


yes 


yes 


no 


0.4764 


0.4935 


0.4619 


0.4375 


0.4092 


0.5086 


0.4207 


0.4624 


yes 


no 


yes 


0.4926 


0.5119 


0.4808* 


0.4698 


0.4599 


0.5499 


0.4638 


0.5170 


no 


yes 


yes 


0.4998* 


0.5301** 


0.4731* 


0.4856** 


0.4421 


0.5618 


0.4383 


0.5171 


yes 


no 


no 


0.4932 


0.4984 


0.4735* 


0.4519 


0.4208 


0.5083 


0.4326 


0.4638 


no 


yes 


no 


0.4931 


0.5084* 


0.4654* 


0.4634* 


0.4085 


0.5134 


0.3945 


0.4554 


no 


no 


yes 


0.4979* 


0.5277** 


0.4673* 


0.4829** 


0.4407 


0.5603 


0.4391 


0.5127 


no 


no 


no* 


0.4711 


0.4884 


0.4436 


0.4448 


0.4009 


0.5069 


0.3884 


0.4469 



(b) Comparison when using the shortest terms method 



Numerical terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average 


precision 


-^■location 


^category 


K detail 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judgc 


B-Judge 


A-Judge 


B-Judge 


yes 


yes 


yes 


0.5012 


0.5205* 


0.4935** 


0.4764 


0.4412 


0.5442 


0.4546 


0.5151 


yes 


yes 


no 


0.4867 


0.4976 


0.4704* 


0.4464 


0.4126 


0.5136 


0.4220 


0.4649 


yes 


no 


yes 


0.5017 


0.5094 


0.4850* 


0.4740 


0.4410 


0.5517 


0.4556 


0.5094 


no 


yes 


yes 


0.4991 


0.5264** 


0.4759* 


0.4841** 


0.4213 


0.5616 


0.4340 


0.5095 


yes 


no 


no 


0.4883 


0.4952 


0.4647* 


0.4444 


0.4247 


0.5076 


0.4200 


0.4614 


no 


yes 


no 


0.4824* 


0.4990* 


0.4537 


0.4509 


0.3927 


0.5119 


0.3901 


0.4517 


no 


no 


yes 


0.4970 


0.5242** 


0.4693* 


0.4804* 


0.4198 


0.5595 


0.4332 


0.5070 


no 


no 


no* 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3850 


0.4468 



0.045 when our extended terms were used. (For exam- 
ple, the average precision of A- Judgement of the short- 
est terms method improved from 0.4488 to 0.4935, i.e., 
0.0447.) This indicates that the extended terms used in 
our experiment were totally effective. Retrieval precision 
can be improved by using location and category infor- 
mation in addition to Robertson's probabilistic retrieval 
method. 

A method that uses one of the extended terms is more 
precise than one using no extended terms. Thus, each ex- 
tended term become effective. The results of the t-test 
show that each extended term has a significant difference 
in at least one evaluation criterion. This indicates that 
location and category information are independently ef- 
fective. 

The main point of our paper is to prove that loca- 
tion information and category information can improve 
the precision of Robertson's probabilistic information re- 
trieval method. This was confirmed by our experimental 
results. 



Use of location information is apt to decrease the pre- 
cision of B- Judgement. This is because B- Judgement 
judges that "a document whose topic is partly relevant to 
a query" is a relevant document. Location information 
weights a term which is in the title or at the beginning of 
the body of a document, i.e., a term which indicates the 
topic of a document. Therefore, for a document where 
the content of a query is written someplace than the topic 
part is not likely to be retrieved. The T-test also showed 
that location information is not significantly different in 
B- Judgement. 

4.3 Effectiveness of detail terms 

This section examines the effectiveness of the terms of 
K detail and Ki engt h- fn our experiments, the shortest 
terms method is used for term extraction. The values of 
the constants of the detail terms are set as in System B 
of Section ||| A comparison of the experimental results is 
shown in Table 0. The four terms K nado , K num , K hira , 



Table 4: Comparison of detailed numerical terms 





Formal run 


Preliminary run 




R-Prceision 


Average precision 


R-Precision 


Average precision 


Detail terms 


A- Judge 


B-Judge 


A- Judge 


B- Judge 


A- Judge 


B- Judge 


A- Judge 


B-Judge 


Neither* 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3850 


0.4468 


Kdeacr Only 


0.4878 


0.5125** 


0.4614* 


0.4674** 


0.4136 


0.5336 


0.3930 


0.4635 


-^proper Ollly 


0.4746 


0.4940 


0.4481 


0.4523 


0.4031 


0.5330 


0.4172 


0.4765 


K nado only 


0.4630 


0.4765 


0.4384 


0.4303 


0.3973 


0.5097 


0.3859 


0.4487 


K„ um only 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3847 


0.4465 


K hira only 


0.4744 


0.4897 


0.4488 


0.4487 


0.3942 


0.5074 


0.3854 


0.4470 


K„ eg Only 


0.4874 


0.5037* 


0.4603 


0.4628* 


0.4019 


0.5134 


0.3967 


0.4554 


K s topword only 


0.4713 


0.4941 


0.4507 


0.4548** 


0.3968 


0.5295 


0.3985 


0.4629 


Ki eng th only 


0.4775 


0.4880 


0.4472 


0.4492 


0.3945 


0.5038 


0.3809 


0.4448 



and K s topword did not improve precision, while Kdescr 
and K neg improved precision greatly. This indicates that 
the following were confirmed by experiments: 

• A term which is obtained from a title of a query 
(DESCRIPTION) is important. 

• A term which is obtained from a expression tagged 
with "NEG" should be removed. 

5 Conclusion 

Our information retrieval method uses Robertson's 2- 
poisson model B, which is one kind of probabilistic ap- 
proach. But, this method does not use location or cate- 
gory information, which should be used to facilitate in- 
formation retrieval. Against this background, we con- 
structed a framework by using location, category and 
detailed information in a 2-poisson model. For the 1999 
IREX contest, we submitted our two systems where their 
precision in the A-judgement measure was 0.4926 and 
0.4827, the highest values among the 15 teams and 22 
systems in the IREX contest. These results indicate that 
our method is comparatively good. 

We carried out comparison experiments in order to 
confirm the effectiveness of each method used in our sys- 
tems. We found that location and category information 
are effective while even the shortest terms method can 
obtain high precision. Also, we found several detailed 
facts such as an expression tagged with "NEG", should 
be removed. 

After this work, by using the technique of IR, we are 
conducting the research on question answering system 
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Abstract 

Robertson's 2-poisson information retrieve model does not use location and category information. We 
constructed a framework using location and category information in a 2-poisson model. We submitted 
two systems based on this framework to the IREX contest, Japanese language information retrieval 
contest held in Japan in 1999. For precision in the A-judgement measure they scored 0.4926 and 
0.4827, the highest values among the 15 teams and 22 systems that participated in the IREX contest. 
We describe our systems and the comparative experiments done when various parameters were 
changed. These experiments confirmed the effectiveness of using location and category information. 

Keyword: 2-poisson model, Location information, Category information 



1 Introduction 

Information retrieval (IR) has become an increasingly im- 
portant area of research due to the rapid growth of the 
Internet. In 1999 the Information Retrieval and Extrac- 
tion Exercise contest (IREX) was held in Japan. We 
submitted two systems to this contest. Their precision 
in the A-judgement measure 1 was 0.4926 and 0.4827, the 
highest values among the 15 teams and 22 systems in the 
IREX contest. This paper describes our systems and the 
comparative experiments done when various parameters 
were changed. 

Our information retrieval method uses Robertson's 2- 
poisson model [8], which is one kind of probabilistic ap- 
proach. But, Robertson's method does not use location 
or category information, which should be used to facili- 
tate information retrieval. Against this background, we 
constructed a framework by using location information, 
category information, and detailed information in a 2- 
poisson model 2 . We verified the effectiveness of using 

1 A-judgment means that a document whose topic is relevant 
to a query is judged a relevant document. 

2 The reason that this paper is entitled "Probabilistic Infor- 
mation Retrieval Using Location Information and Category Infor- 
mation" is that our methods use location and category informa- 
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these three types of information by doing comparative 
experiments. When the 2-poisson model is used a term 
extraction method needs to be selected. In this paper, 
we describe four term extraction methods, and compared 
them in experiments. 

2 Information retrieval 
2.1 Task 

The information retrieval tasks in this paper are identical 
to those for the IREX contest. The database used for 
information retrieval (the same used in IREX) is from 
two-years (1994-1995) of a Japanese newspaper [3]. We 
retrieved from this database documents which satisfied 
the information condition for a Japanese language query. 

The following is an example of a query. (The data is 
from the IREX preliminary experiment.) 

Example of a Japanese query 

<T0PIC> 

<TOPIC-ID>100K/TOPIC-ID> 
<DESCRIPTI0N> £ 31 £ f# </DESCRIPTlON> 
<NAHRATIVE>fE* : t ttife^-g-fffilc^ffllSS^a^ h tl 

ifcslHiKti-tt?. </MARRATIVE> 
</T0PIC> 

English translation 

tion in addition to Robertson's probabilistic information retrieval 
method. 



<TOPIC> 

<T0PIC-ID>100K/T0PIC-ID> 
<DESCRIPTION>enterprise amalgamation 
</DESCRIPTIOM> 
<NARRATIVE> 

The condition for a relevent document is 
that in the document an announcement of 
enterprise amalgamation materialization is 
described and the name of the enterprise 
which participated the amalgamation can be 
recognized. Also, one of the field of 
amalgamation enterprise and its purpose 
should be able to recognized. Enterprise 
amalgamation contains enterprise annexation, 
enterprise integration, and enterprise 
purchasing. </NARRATIVE> 
</T0PIC> 

The number indicated by <T0PIC-ID> means the ID 
number of the query. <DESCRIPTION> contains a phrase 
that indicates the information needed. <NARRATTVE> 
contains the sentences that restrict the information re- 
quested. During the task, the system receives a query 
such as the above one and outputs 300 documents in 
order of confidence. 

Example of a Japanese document 

<DOCNO>95O217O9K/D0CN0> 
<SECTI0N> SSr</SECTI0N> 

"ft </HEADLINE> 
<TEXT> 

r, #*t ■ ;wtp, m\m±m s-oo%^aftt 

</TEXT> 
</D0C> 

English translation 

<DOCNO>95O217O9K/D0CN0> 
<SECTION>Economic page</SECTI0N> 
<HEADLINE>Tounen company completely makes 
Kigunasu company a subsidiary company 
</HEADLINE> 
<TEXT> 

Tounen company anounced that it makes 
Kigunasu comapry, one of its group companies, 
(The capital is one billion yen. The head 
office is in Kawasaki city. The president 
is Mr. Toshihide Mori.) a completely 
subsidiary company. Kigunasu company is 
invested 70'/, by Tounen company and 30'/, by 
Nichimou company, and Tounen company 
purchases 600,000 stocks which Nichimou 
company possesses at 12,500,000,000 yens. 
</TEXT> 
</D0C> 

In this document, the newspaper information cate- 
gory (the economic or political pages) is indicated by 
<SECTI0N>, the title of the document is indicated by 
<HEADLINE>, and the body of the document is indicated 
by <TEXT>. The tool "trec_eval" of TREC is used to 
evaluate the retrieval results [12]. In the contest, "R- 
Precision" were used. It indicates the precision when re- 
trieving R documents, where R is the number of relevant 
documents. 



2.2 Outline of our method 

Our information retrieval method uses Robertson's 2- 
poisson model [8] which is one kind of probabilistic ap- 
proach. Robertson's method calculates each document's 
score using the following equation 3 , then outputs the doc- 
uments with high scores as retrieval results. (The follow- 
ing Score(d,q) is the score of a document d against a 
query q.) 



Score(d, q) 



term t 
in q 



tf(d,t) x N 

length(d) df(t) 



tf(d,t) + k t 



. tf q (q,t) ' 
tf q (q,i) + kq , 



(1) 



where terms occur in a query. tf(d, i) is the frequency of 
a term t in a document d, tf q (g, i) is the frequency of t in 
a query q, df(t) is the number of the documents in which 
t occurs, N is the total number of documents, length(d) 
is the length of a document d, and A is the average length 
of the documents, kt and k q are constants which are set 
by experiments. 



In this equation, we call 



tf(d,t) + k t 



length(d) 



the TF 



term, (abbr. TF{d,t)), log^ffa the IDF term, (abbr. 
IDF(t)), and theTF, term, (abbr. TF q (q,t)). 

Our method adds several extended terms to this equa- 
tion, and is expressed by the following equation. 



Score(d.q) = K category (d) 1 ^ (TF(d,t) X IDF(t) 

term t 
k in q 

XTF q (q,t)X K detail (d,t) X K location (d,t)) 
length(d) \ 



+ 



length(d) + A _ 



(2) 



The TF, IDF and TF 9 terms in this equation are 
identical to those in Eq. (1). The term ie '„ e "th+A nas 
a higher value when a document is longer. This term is 
made because if the other information is exactly equal, 
the longer document is more likely to include the content 

requested by the query. Kcategoryi ^-detail and ^location 

are extended numerical terms made to improve precision. 
Kcategory uses the category information of the document 
found in newspapers, such as the economic and political 
pages. Kiocation uses the location of the term in the doc- 
ument. If a term is in the title or at the beginning of 
the body of the document, it is given a higher weighting. 
Kdetaii uses the information such as whether the term is 
a proper noun and or a stop word such as JC9 "docu- 
ment" and % © "thing" . In the next section, we explain 
these extended numerical terms in detail. 

2.3 Extended numerical terms 

We use the three extended numerical terms of Ki oca tiom 
Kcategory, and Kdetaii as in Eq. (2). This section explains 
them in detail. 



This equation is BM11, which corresponds to BM25 in the 
case of b = 1[9]. Although we made experiments testing some 
cases of b in BM25, the case of b = 1 was roughly better than any 
other cases in this work. So we used BM11. 



1. Location information {Kiocation) 

In general, the title or the first sentence of the body 
of a document in a newspaper very often indicates 
its subject. Therefore, the precision of information 
retrieval can be improved by weighting the terms in 
these two locations. The term Ki oca ucm performs 
this task, and changes the weight of a term based 
its location at the beginning of the document. If a 
term is in the title or at the beginning of the body, 
it is given a high weighting. Otherwise, it is given 
low weighting. This term is expressed as follows: 



Kloc 



,(d, t) : 



^location, 1 

(when a term t occurs in the title of 
a document d), 



ocatioTi ? "2 

(otherwise) 



(length(d) - 2 * P(d, t)) 
length(d) 



(3) 



P(d, t) is the location where a term t occurs in the 
document d. When a term occurs more than once in 
a document, its first occurrence is used. ltiocation,\ 
and kiocation,2 are constants which are set by ex- 
periments. 

2. Category information (K ca tegory) 

Kcategory uses category information such as the 
economic and political pages. This functions as a 
technique called relevance feedback [10]. First, we 
specify the categories which occur in the top 100 
documents of the first retrieval when K ca t e gory — 1- 
Then, we increase the scores of documents hav- 
ing the same categories. For example, if economic 
pages often occur in the top 100 documents of 
the first retrieval, we increase the score of a docu- 
ment whose page is a economic page and decrease 
the score of the document whose page is different. 
Kcategory is expressed as follows; 



^■category (,d) — 1 ~f" ^category 



(RatioA(d) - RatioB(d)) 
(RaUoA(d) + RatioB(d)) 



(4) 

where RatioA is the ratio of a category in the top 
100 documents of the first retrieval. RatioB is the 
ratio of a category in all the documents. The value 
of Kcategory (d) is large, when RatioA is large (page 
of a document d occurs frequently in the top 100 
documents of the first retrieval.) and RatioB is 
small (page of a document d does not occur often 
in all the documents.), kcategory is a constant which 
is set by experiments. 

3. Other information {Kdetaii) 

Kdetaii is a more detailed numerical term that uses 
different information, such as whether the term is 
a proper noun and whether the term is a stop word 
such as A It "document" and * © "thing" . If a 
term is a proper noun, it is weighted high. If a term 
is a stop word, such as • "document" and % © 
"thing," it is weighted low. Kdetaii is expressed as 
follows for simplicity, the variables for a document 
and a term, d and t, are omitted: 



^detail — ^descr ' 

Each term in this equation is explained below. 



(5) 



K d escr 

When a term is obtained from the title of a 
query, i.e. DESCRIPTION, K deS cr = k deS cr(> 
1). Otherwise, Kdescr — 1. This is because 
a term obtained from the title of a query is 
important. 

K, 



K, 



proper — 

1. This 



proper 

When a term is a proper noun, 

kproper (-** 1). Otherwise Kproper 

is because a term that is a proper noun is im- 
portant. 

Knado 

When a term is followed by the Japanese word 
nado (such as) in a query sentence, K na do — 
k„ado(> !)• Otherwise K nado = 1. A term 
which is followed by the Japanese word nado 
is specific in meaning and is just as important 
as a proper noun. 

Knum 

When a term is numeric, K num — k num {< 1). 
Otherwise, K num — 1. A term which consists 
of only numerals does not contain much rele- 
vant information making it unimportant to a 
query. 

Khira 

When a term consists of hiragana characters 
only, K hira - k hira (< 1). Otherwise, K hira 
— 1. A term which consists of only hiragana 
characters does not contain much relevant in- 
formation making it unimportant to a query. 



• K n 



When a term is obtained from a region tagged 
with a NEG tag in a query, K neg — k„ eg . Oth- 
erwise K neg — 1. 

In a query of the IREX contest, an expression, 
"... wa nozoku" (... is excepted), as in the 
following query, was tagged with a NEG tag. 

Example Japanese query 
<T0PIC> 

<T0PIC-ID>lOO3</T0PIC-ID> 
<DESCRIPTI0N> H » * © W. it </DESCRIPTI0N> 
<MARRATIVE>¥ft*l#M»* f H 5t ffl }g » 

(C » & SMW-<DW.MK-d V TM^ e> tlX V 

iW^WKfr&lto <NEG> H*fflgWSB 

5 iS * It B< o </NEGX/NARRATIVE> 
</T0PIC> 

English translation 
<T0PIC> 

<T0PIC-ID>iOO3</T0PIC-ID> 
<DESCRIPTI0N>Dispatch of the United 
Nations f orces</DESCRIPTI0N> 
<NARRATIVE>The condition for a 
relevent document is that in the 
document a dispatch of the United 
Nations forces in the activity of UN 
such as peace maintainence activity 
is described. The purpose of the 
dispatch or the target region should 
be described. <NEG>A document 
describing the discussion of whether 
the Self-Defense Forces of Japan is 
dispatched to UN or not is elimated. 



</NEGX/NARRATIVE> 
</TOPIC> 

If a term from a region tagged with a NEG tag 
is used, non-relevant documents are often re- 
trieved and therefore such a term is weighted 
low. In this paper, k„ eg is set to 0. This indi- 
cates that a term from a region tagged with a 
NEG tag is not used in retrieval. 

• K-stopword 

When a term is a stopword such as jouken 
(condition), kiji (document) and baai (case), 

K stopword — kstopw or ■d( < Z. 1)> Otherwise Kstopword 

— 1. A term that is a stopword is unimportant. 
Each constant, such as kdescr, is set experimentally. 

2.4 How to extract terms 

Before being able to use Eq. (2) in information retrieval, 
we must extract the terms from a query. This section 
describes how to do this. With regard to term extraction, 
we considered the several methods listed below. 

1. Method using only the shortest terms 

This is the simplest method. The method divides 
the query sentence into short terms by using the 
morphological analyzer "juman" [2] and eliminates 
non-nominal words and stop words 4 . The remain- 
ing words are used in the retrieval process. 

2. Method using all term patterns 

In the first method the terms are too short. For 
example, "enterprise" and "amalgamation" are 
used instead of "enterprise amalgamation." 5 We 
thought that we should use "enterprise amalgama- 
tion" in addition to the two short terms. Therefore, 
we decided to use both short and long terms. We 
call this "all-term patterns method." For example, 
when "enterprise amalgamation materialization" 
was inputted, we use "enterprise", "amalgama- 
tion", "materialization", "enterprise amalgama- 
tion" , "amalgamation materialization" , and "en- 
terprise amalgamation materialization" as terms for 
information retrieval. We thought that this method 
would be effective because it uses all term patterns. 
But, we also thought that it is inequitable that only 
the three terms of "enterprise," "amalgamation," 
"materialization," are derived from "... enterprise 
... amalgamation ... materialization while on 
the other hand six terms are derived from "en- 
terprise amalgamation materialization." We exam- 
ined several normalization methods in preliminary 
experiments, an d decid ed to divide the weight of 

each term by ^ re ( re 2 +1 ) ; where n is the number of 

successive words. For example, in the case of "en- 




amalgamation materialization 



amalgamation ^materialization 



enterprise amalgamation 




enterprise amalgamation materialization 

Figure 1: An example of a lattice structure 



terprise amalgamation materialization" , n — 3. 
Method using a lattice 

Although the method using all-term patterns is ef- 
fective for use with all patterns of terms, it needs to 



4 Since, Japanese is an agglutinative languages like Chinese, 
there are no spaces between words and a morphological analyzer 
is necessary to divide a sentence into words. 

5 Although this paper deals only with Japanese, not English, for 
this explanation we use English examples for the English readers. 
This method handles compound nouns and can be used not only 
for Japanese but also for English. 



be normalized by using the adhoc equation \J n ^ n £ V> . 

Thus, we considered the method where all the term 
patterns are stored into a lattice structure. We use 
the patterns in the path where the score in Eq. 
(2) is the highest. (This method is almost same as 
Ozawa's [7]. The differences are the fundamental 
equation for information retrieval, and whether to 
use or not use a morphological analyzer.) 

For example, in the case of "enterprise amalgama- 
tion materialization" a lattice, as shown in Fig. 
1, is obtained. As in this figure, four paths exist 
where each of their scores are calculated by Eq. (2) 
and the terms in the highest path are used. This 
method does not require the adhoc normalization 
as in the method using all the term patterns. 

4. Method using down-weighting [1] 

This is the method that Fujita proposed at the 
IREX contest, and we examined after the contest. 
It is similar to the all-term patterns method. It 
uses all the term patterns but the normalization 
is different from the all-term patterns method. It 
does not change the weight of the shortest terms; 
and decreases the weight of the longer terms. We 
decided to multiply the weight kd OW n X ~ 1 to a term, 
when it consisted of x shortest terms, where kdown 
was set by experiments. This method basically uses 
the shortest terms while also using the longer terms 
by down-weighting them. 

3 IREX contest results 

For our two submissions to the IREX contest 6 , we se- 
lected the "all-term patterns" and "lattice structure" 
methods to extract terms 7 , and set the constants of the 
extended terms in order to maximize the precision in the 
preliminary-run data as follows. 

1. System A 

It used the lattice method for the term extraction. 
The parameters were set as follows; ki oca tion,i — 

1.35, klocation,2 — 0.125, kcategory — 0, kdescr — 1.5, 
kproper — 2, knado — 1, knum — 0.5, khira — 0.5, 
kneg — 0, kstopword,! — 0, kstopword,2 — 0.5, kt — 1, 



IREX allowed two systems to be submitted. 
7 The reason we did not use "the shortest terms method" is 
because it is too simple and did not seem effective. The "down- 
weighting method" is a method proposed at IREX. So we could 
not use it in IREX. 



Table 1: R-Precision of all the systems 



System ID 


A-Judgment 


B-Judgment 


1103a 


0.4505 


0.4888 


1103b 


0.4657 


0.5201 


1106 


0.2360 


0.2120 


1110 


0.3329 


0.4276 


1112 


0.2790 


0.3343 


1120 


0.2713 


0.3339 


1122a 


0.3808 


0.4689 


1122b 


0.4034 


0.4747 


1126 


0.0966 


0.0891 


1128a 


0.3384 


0.3897 


1128b 


0.3924 


0.4175 


1132 


0.0602 


0.0791 


1133a 


0.2383 


0.2277 


1133b 


0.2457 


0.2248 


1135a 


0.4926 


0.5119 


1135b 


0.4827 


0.4878 


1142 


0.4455 


0.4929 


1144a 


0.4658 


0.5510 


1144b 


0.4592 


0.5442 


1145a 


0.3352 


0.3424 


1145b 


0.2553 


0.2935 


1146 


0.2220 


0.2742 



and k q = 0.1. Terms obtained from DESCRIP- 
TION are handled as terms different from terms 
obtained from NARRATIVE. 

2. System B 

It used all-term patterns method for term extrac- 
tion. The parameters were set as follows; ki OC aUon,i — 

1.3, klocation,'! — 0.15, k ca tegory — 0.1, kdescr — 
1.75, hproper — 2, hnado — 1*7, knum — 0.5, hhira — 
0.5, kneg — 0, k s topword,\ — 0, k s topword,2 — 0.5, 

kt — 1, and k q — 0. Terms obtained from DE- 
SCRIPTION were handled as the terms different 
from those obtained from NARRATIVE. 

In the contest, the results for the 22 systems were sub- 
mitted by the 15 teams. Their R-Precisions are shown 
in Table 1. The first column of the table indicates the 
names of the systems. Our two systems, System A and 
System B correspond to 1135a and 1135b. A-Judgement 
and B-Judgement are the evaluation criteria determined 
by the IREX committee. A-Judgment means that a doc- 
ument whose topic is relevant to a query is judged as a 
relevant document. B- Judgment means that a document 
whose topic is partly relevant to a query is also judged as 
a relevant document. Although our systems were not the 
highest in B-Judgment, they were the highest among all 
the systems in A-Judgment. This result indicates that 
our method is relatively superior. 

4 Experiments 

In this section we describe several experiments done to 
test the effectiveness of the several methods used in our 
system. In the experimental results of this section, we 
also show Average Precision (the average of the precision 
when each relevant document is retrieved) in addition to 
R-Precision. For the comparison experiments, t-test is 
used. A method tagged with "#" in Tables 2 to 4) is 
the base for comparison. A method tagged with "*" is 
superior to the base method at the significance level of 



5%, and a method tagged with "**" is superior at the 
significance level of 1%. T-test is used only in formal- 
run experiments. (The preliminary-run data contained 
six queries, and the formal-run data contained thirteen 
queries.) 

4.1 Comparison of term extraction methods 

We showed the following four term extraction methods 
in Section 2.4. 

1. Method using the shortest terms 

2. Method using all the term patterns 

3. Method using a lattice 

4. Method using down-weighting 

All the comparison results are shown in Table 2. In 
Table 2(a) all extended terms were used. In Table 2(b) 
no extended terms were used. In the down-weighting 
method we tested the two cases of kdown — 0.1 and 

kdown — 0.01. 

The precision of the all-term patterns method was 
lowest in the formal run. It needed to be normalized us- 
ing the adhoc equation. Since it had the lowest precision, 
it was thought to be inferior to the other methods. Also, 
it was shown by t-test to be significantly inferior to the 
shortest terms method. 

Although the down-weighting method obtained the 
highest precision when no extended terms were used, it 
was not as effective when all the extended terms were 
used. Since it was significantly different from any of the 
other methods, cannot say that it is very reliable. But, 
in the case where a small amount of retrieval information 
was used (i.e. no extended terms) it was very effective. 

Since only the shortest terms method is significantly 
different from the all-term patterns method, we think it is 
a sound method which can provide reliable results. Since 
the lattice and the down-weighting methods are not sig- 
nificantly different from the all-term patterns method, we 
think that they must have some problems. One problem 
that occurred when using the lattice method was that 
the terms used in retrieval easily changed depending on 
the context, while the down-weighting method's problem 
was that it uses the extra terms even if it down-weights 
them. However, it is thought by us that using longer 
terms in addition, is better than using only the short- 
est terms. We have to continue the investigation of term 
extractions. 

4.2 Effectiveness of extended terms 

Extended terms used in this paper are classified into the 
following three categories: 

1. Kiocation (location information) 

2. K ca tegory (category information) 

3. Kdetaii (detail information) 

(Here, K detail contains K len(j th — Jngth+b whicn 
is the numerical term for a document length in Eq 
(2)-) 

In order to verify the effectiveness of the above three 
extended terms, we carried out eight experiments in 
which these three terms were alternately used or not 



Table 2: Comparison of methods to extract keywords 
(a) When all extended terms are used 



Method to extract terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average precision 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


Using the shortest terms 


0.5012 


0.5205** 


0.4935** 


0.4764* 


0.4412 


0.5442 


0.4546 


0.5151 


Using all term patterns* 


0.4827 


0.4878 


0.4553 


0.4453 


0.4373 


0.5573 


0.4576 


0.5317 


Using the lattice structure 


0.4926 


0.5119 


0.4808 


0.4698 


0.4599 


0.5499 


0.4638 


0.5170 


Using down-weight (kdown = 0.01) 


0.5006 


0.5217 


0.4935 


0.4778 


0.4412 


0.5445 


0.4546 


0.5157 


Using down-weight (kdo-wn = 0.1) 


0.4997 


0.5233 


0.4939 


0.4809 


0.4478 


0.5504 


0.4563 


0.5185 



(b) When no extended terms are used 



Method to extract terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average precision 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


Using the shortest terms 


0.4744 


0.4897 


0.4488* 


0.4487* 


0.3900 


0.5082 


0.3850 


0.4468 


Using all term patterns* 


0.4445 


0.4665 


0.4172 


0.4180 


0.3965 


0.4981 


0.3960 


0.4444 


Using the lattice structure 


0.4711 


0.4884 


0.4436 


0.4448 


0.4009 


0.5069 


0.3884 


0.4469 


Using down-weight (kdown = 0.01) 


0.4760 


0.4896 


0.4492 


0.4494 


0.3940 


0.5082 


0.3850 


0.4470 


Using down- weight (kdown = 0.1) 


0.4816 


0.4986 


0.4545 


0.4568 


0.4003 


0.5076 


0.3860 


0.4498 



The method tagged with "#" is a base method for comparison. A result tagged with "*" is superior to the base 
method's at the significance level of 5%, and a result tagged with "**" is superior at the significance level of 1%. 



Table 3: Comparison of extended numerical terms 
(a) Comparison when using the lattice method 



Numerical terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average precision 


^■location 


Kcategory Kdetail 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


yes 


yes 


yes 


0.5031 


0.5161 


0.4888* 


0.4745 


0.4495 


0.5471 


0.4625 


0.5202 


yes 


yes 


no 


0.4764 


0.4935 


0.4619 


0.4375 


0.4092 


0.5086 


0.4207 


0.4624 


yes 


no 


yes 


0.4926 


0.5119 


0.4808* 


0.4698 


0.4599 


0.5499 


0.4638 


0.5170 


no 


yes 


yes 


0.4998* 


0.5301** 


0.4731* 


0.4856** 


0.4421 


0.5618 


0.4383 


0.5171 


yes 


no 


no 


0.4932 


0.4984 


0.4735* 


0.4519 


0.4208 


0.5083 


0.4326 


0.4638 


no 


yes 


no 


0.4931 


0.5084* 


0.4654* 


0.4634* 


0.4085 


0.5134 


0.3945 


0.4554 


no 


no 


yes 


0.4979* 


0.5277** 


0.4673* 


0.4829** 


0.4407 


0.5603 


0.4391 


0.5127 


no 


no 


no# 


0.4711 


0.4884 


0.4436 


0.4448 


0.4009 


0.5069 


0.3884 


0.4469 



(b) Comparison when using the shortest terms method 



Numerical terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average precision 


^location 


Kcategory Kd e t a n 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


yes 


yes 


yes 


0.5012 


0.5205* 


0.4935** 


0.4764 


0.4412 


0.5442 


0.4546 


0.5151 


yes 


yes 


no 


0.4867 


0.4976 


0.4704* 


0.4464 


0.4126 


0.5136 


0.4220 


0.4649 


yes 


no 


yes 


0.5017 


0.5094 


0.4850* 


0.4740 


0.4410 


0.5517 


0.4556 


0.5094 


no 


yes 


yes 


0.4991 


0.5264** 


0.4759* 


0.4841** 


0.4213 


0.5616 


0.4340 


0.5095 


yes 


no 


no 


0.4883 


0.4952 


0.4647* 


0.4444 


0.4247 


0.5076 


0.4200 


0.4614 


no 


yes 


no 


0.4824* 


0.4990* 


0.4537 


0.4509 


0.3927 


0.5119 


0.3901 


0.4517 


no 


no 


yes 


0.4970 


0.5242** 


0.4693* 


0.4804* 


0.4198 


0.5595 


0.4332 


0.5070 


no 


no 


no# 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3850 


0.4468 



used. These experiments were performed using "the lat- 
tice method" and "the shortest terms method". The 
results are shown in Table 3. 

The last line of the table is the case where no extended 
terms were used and the first line of the table is the case 
where they were all used. When we compared the two 
lines, we found there was an improvement of 0.027 to 
0.045 when our extended terms were used. (For exam- 
ple, the average precision of A-Judgement of the short- 
est terms method improved from 0.4488 to 0.4935, i.e., 
0.0447.) This indicates that the extended terms used in 
our experiment were totally effective. Retrieval precision 
can be improved by using location and category infor- 
mation in addition to Robertson's probabilistic retrieval 
method. 

A method that uses one of the extended terms is more 
precise than one using no extended terms. Thus, each ex- 
tended term become effective. The results of the t-test 
show that each extended term has a significant difference 
in at least one evaluation criterion. This indicates that 



location and category information are independently ef- 
fective. 

The main point of our paper is to prove that loca- 
tion information and category information can improve 
the precision of Robertson's probabilistic information re- 
trieval method. This was confirmed by our experimental 
results. 

Use of location information is apt to decrease the pre- 
cision of B-Judgement. This is because B-Judgement 
judges that "a document whose topic is partly relevant to 
a query" is a relevant document. Location information 
weights a term which is in the title or at the beginning of 
the body of a document, i.e., a term which indicates the 
topic of a document. Therefore, for a document where 
the content of a query is written someplace than the topic 
part is not likely to be retrieved. The T-test also showed 
that location information is not significantly different in 
B-Judgement. 



Table 4: Comparison of detailed numerical terms 



Detail terms 


Formal run 


Preliminary run 


R-Precision 


Average precision 


R-Precision 


Average precision 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


A-Judge 


B-Judge 


Neither* 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3850 


0.4468 


Kdescr only 


0.4878 


0.5125** 


0.4614* 


0.4674** 


0.4136 


0.5336 


0.3930 


0.4635 


^■■proper Only 


0.4746 


0.4940 


0.4481 


0.4523 


0.4031 


0.5330 


0.4172 


0.4765 


Knado only 


0.4630 


0.4765 


0.4384 


0.4303 


0.3973 


0.5097 


0.3859 


0.4487 


K„ um only 


0.4744 


0.4897 


0.4488 


0.4487 


0.3900 


0.5082 


0.3847 


0.4465 


Khira only 


0.4744 


0.4897 


0.4488 


0.4487 


0.3942 


0.5074 


0.3854 


0.4470 


Kneg Only 


0.4874 


0.5037* 


0.4603 


0.4628* 


0.4019 


0.5134 


0.3967 


0.4554 


Kstopword only 


0.4713 


0.4941 


0.4507 


0.4548** 


0.3968 


0.5295 


0.3985 


0.4629 


Ki e „gth only 


0.4775 


0.4880 


0.4472 


0.4492 


0.3945 


0.5038 


0.3809 


0.4448 



4.3 Effectiveness of detail terms 

This section examines the effectiveness of the terms of 
Kdetaii and Ki en gth- In our experiments, the shortest 
terms method is used for term extraction. The values of 
the constants of the detail terms are set as in System B 
of Section 3. A comparison of the experimental results is 
shown in Table 4. The four terms K na do, K num , Khira-, 
and K s topword did not improve precision, while Kdescr 
and K neg improved precision greatly. This indicates that 
the following were confirmed by experiments: 

• A term which is obtained from a title of a query 
(DESCRIPTION) is important. 

• A term which is obtained from a expression tagged 
with "NEG" should be removed. 

5 Conclusion 

Our information retrieval method uses Robertson's 2- 
poisson model [8], which is one kind of probabilistic ap- 
proach. But, this method does not use location or cate- 
gory information, which should be used to facilitate in- 
formation retrieval. Against this background, we con- 
structed a framework by using location, category and 
detailed information in a 2-poisson model. For the 1999 
IREX contest, we submitted our two systems where their 
precision in the A-judgement measure was 0.4926 and 
0.4827, the highest values among the 15 teams and 22 
systems in the IREX contest. These results indicate that 
our method is comparatively good. 

We carried out comparison experiments in order to 
confirm the effectiveness of each method used in our sys- 
tems. We found that location and category information 
are effective while even the shortest terms method can 
obtain high precision. Also, we found several detailed 
facts such as an expression tagged with "NEG" , should 
be removed. 

After this work, by using the technique of IR, we are 
conducting the research on question answering system 
[6]. 
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